Method and apparatus for a panoramic camera to capture a 360 degree image

ABSTRACT

Most camera systems only record an image from a limited viewing angle. A new panoramic camera apparatus is disclosed that instantaneously captures a 360 degree panoramic image. In the camera device, virtually all of the light that converges on a point in space is captured. Specifically, in the camera of the present invention, light striking this point in space is captured if it comes from any direction, 360 degrees around the point and from angles 50 degrees or more above and below the horizon. The panoramic image is recorded as a two dimensional annular image. Furthermore, various different systems for displaying the panoramic images and distributing the panoramic images. Specifically, methods and apparatus for digitally performing a geometric transformation of the two dimensional annular image into rectangular projections such that the panoramic image can be displayed using conventional methods such as printed images and televised images.

This application claims the benefit of the U.S. provisional patentapplication with Ser. No. 60/020,292 that was filed on Jun. 24, 1996.

FIELD OF THE INVENTION

The present invention relates to the field of film and videophotography. In particular the present invention discloses a cameradevice that captures a 360 degree panoramic image and display systemsfor displaying the panoramic image captured by the camera device.

BACKGROUND OF THE INVENTION

Most cameras only provide a small viewing angle. Thus, a typicalconventional camera only captures an image in the direction that thecamera is aimed. Limited view cameras force viewers to look only at whatthe camera operator chooses to focus on. Some cameras use a specializedwide angle lens to capture a wider panoramic image, but such panoramiccameras still have a limited field of view.

It would be desirable to have a camera system that would capture thelight from all directions such that a full 360 degree panoramic imagecan be created. A full 360 degree panoramic image would allow the viewerto choose what she would like to look at. Furthermore, a full 360 degreepanoramic image allows multiple viewers to simultaneously view the worldfrom the same point, with each being able to independently choose theirviewing direction and field of view.

At the present time, there are some known methods of creating 360 degreepanoramic images. However, most current methods are subject tolimitations due to their physical movements and mechanical complexity.For example, some of the current methods operate by combining a seriesof individual photographs taken in different directions into a singlepanoramic image. Some panoramic cameras spin a lens and film to capturea panoramic view in a single sweeping motion.

There is a market for panoramic photos to be used in multimediaapplications, typically provided on CD-ROMs. In the last few years, somesoftware manufacturers have introduced standards for digital storage andcomputer playback of panoramic datasets. One example is QuickTime® VR,introduced by Apple® Computer, Inc. Apple® Computer's QuickTime® VRstandard governs the file storage format and the playback softwareneeded to view their datasets.

Currently, Apple Computer recommends and provides software tools toimplement a labor-intensive process for capturing these panoramicdatasets. In the Apple QuickTime® VR (QTVR) process a standard 35 mmcamera is mounted vertically on a leveled tripod and equipped with anextreme wide angle lens (e.g. 15-18 mm focal length). A sequence oftwelve or more overlapping still photographs is taken at roughly 30degree intervals as the camera is turned on the tripod around a verticalaxis. These photographs are developed, digitized and then fed into asemi-automated software program called a “stitcher” that merges theoverlapping still photographs into one long panoramic strip.

The labor intensive process suffers from a number of shortcomings.First, the process is time-consuming since many steps require humanintervention and guidance. Furthermore, the recommended process is proneto temporal artifacts since it captures each individual photo at adifferent time. This means that the “stitched” pan image is notinstantaneous but rather is made up of individual photos taken atdifferent times. The time change during the series of photographs makesit nearly impossible to create panoramic images in changing scenescontaining shorelines, urban crowds and traffic, windblown trees, etc.Finally, it is difficult to see how the image capture method recommendedby Apple QuickTime® VR (QTVR) can be extended from a single stillpanoramic image into a continuous frame, or motion picture panoramicimage capture.

SUMMARY OF THE INVENTION

The present invention discloses a camera device that instantaneouslycaptures a 360 degree panoramic image. Furthermore, the presentinvention discloses various different systems for displaying thepanoramic images.

In the camera device, virtually all of the light that converges on apoint in space is captured. Specifically, in the camera of the presentinvention, light striking this point in space is captured if it comesfrom any direction, 360 degrees around the point and from angles 50degrees or more above and below the horizon as illustrated in FIG. 1.

Other objects, features and advantages of present invention will beapparent from the company drawings and from the detailed descriptionthat follows below.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will beapparent to one skilled in the art, in view of the following detaileddescription in which:

FIG. 1 illustrates the panoramic surroundings that are captured by thepanoramic camera system of the present invention.

FIG. 2a illustrates a schematic diagram of the panoramic camera systemof the present invention.

FIG. 2b illustrates a schematic diagram of the panoramic camera systemof the present invention with a parabolic mirror.

FIG. 2c illustrates an annular image captured by the panoramic camerasystem of the FIG. 2b wherein the incident angle is linearlyproportional to the radial distance of the annular image.

FIG. 3a illustrates an example of an annular image captured by thepanoramic camera system of the present invention.

FIG. 3b illustrates a rectangular panoramic image after the capturedannular image is transformed from polar coordinates to rectangularcoordinates.

FIG. 4a illustrates photographic film used to capture the annularpanoramic image.

FIG. 4b illustrates a Charged Coupled Device array used to capture theannular panoramic image.

FIG. 5 illustrates an alternate embodiment of the camera system of thepresent invention wherein a beam splitter is used to allow the annularimage to be captured on two image planes.

FIG. 6a illustrates a first embodiment of two image planes used tocapture different portions of a single annular panoramic image.

FIG. 6b illustrates a second embodiment of two image planes used tocapture different portions of a single annular panoramic image.

FIG. 7 illustrates an embodiment of the panoramic camera wherein some ofthe optical elements are housed within the parabolic mirror.

FIG. 8a illustrates a first embodiment of the panoramic camera that usesa solid transparent block to surround the parabolic mirror.

FIG. 8b illustrates a second embodiment that uses a solid transparentblock to surround the parabolic mirror which houses other opticalelements.

FIG. 9a illustrates an embodiment that panoramic camera that supportsthe convex mirror with a central post that is out of the annular fieldof view.

FIG. 9b illustrates an embodiment that panoramic camera that divides theconvex mirror in quarters and supports the mirror using posts betweenthe four quarters.

FIG. 10 graphically illustrates how the annular image is sampled toproduct a rectangular panoramic image.

FIG. 11a graphically illustrates how an image is stored in Apple®Computer's QuickTime® VR format.

FIG. 11b graphically illustrates how viewports are created from Apple®Computer's QuickTime® VR format.

FIG. 11c illustrates a flow chart that lists how the panoramic camerasystem can be used to create images in Apple® Computer's QuickTime® VRformat.

FIG. 12a illustrates a graphical user interface for a client programused to view panoramic still images created by the panoramic camerasystem.

FIG. 13a illustrates a graphical user interface for a client programused to view panoramic video created by the panoramic camera system.

FIG. 13b illustrates a networked computer arrangement used to viewpanoramic video created by the panoramic camera system.

FIG. 14a illustrates a side view of one embodiment of the panoramiccamera system that includes microphones.

FIG. 14b illustrates a side view of one embodiment of the panoramiccamera system that includes microphones.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and apparatus for a camera device that instantaneously captures360 degree panoramic images is disclosed. In the following description,for purposes of explanation, specific nomenclature is set forth toprovide a thorough understanding of the present invention. However, itwill be apparent to one skilled in the art that these specific detailsare not required in order to practice the present invention. Forexample, the present invention has been described with reference toCharge Coupled Devices. However, the panoramic camera system can easilybe implemented with other types of electronic image capture systems.

The Basic Panoramic Camera Design

The panoramic camera design of the present invention captures light fromall directions within 50 to 60 degrees above and below the horizonsimultaneously. FIG. 1 graphically illustrates the cylindrical panoramicview of which the panoramic camera system captures an image. To captureall the light of the panorama and generate a two dimensionalrepresentation that may easily be recorded, the present invention uses acarefully designed and engineered collection of mirrors and lenses. Thebasic design of the panoramic camera of the present invention isillustrated in FIG. 2a. Each element of the panoramic camera will bedescribed individually.

The Mirror

Referring to FIG. 2a, the panoramic camera of the present inventioncollects light using a convex mirror 210 that is in the approximateshape of parabolic cone. In one embodiment of the present invention, thetip of the convex mirror 210 is pointed downward as illustrated in FIG.2a. When the convex mirror 210 is viewed from below, the parabolicmirror 210 presents an annular image of the surrounding panorama asillustrated in FIG. 3a. However, the annular image is distorted andsuffers from optical defects due to the shape of the convex mirror 210.

The distortion in the image is partly due to the fact that the convexmirror 210 of the imaging system effectively converts the surroundingpanorama to a polar coordinate system. By adjusting the shape of theconvex mirror 210, the mapping of the elevation angle of incoming lightto radial distance in the annular image, can be controlled.

In a preferred embodiment, the convex mirror 210 is a parabolic mirrorthat creates an annular image wherein the radial distance from thecenter of the annular image is linearly proportional to the angle ofincident light. A panoramic camera system with a parabolic mirror isillustrated in FIG. 2b. Note that in the image plane of FIG. 2b, thedistance from the center is linearly proportional to the angle ofincident light. This concept is more clearly illustrated in FIG. 2c,wherein the concentric circles represent different angles of incidentlight.

The Astigmatism Correction Lens

The convex mirror of the present invention introduces other imagedefects that require careful correction. One particular problem isastigmatism. Specifically, the light reflected downward from the convexmirror 210 of the present invention will not meet at a single focalpoint. To correct for this problem, an astigmatism correction lens 220is added to correctly focus the light from the convex mirror 210.

The astigmatism correction lens 220 comprises a group of 2 or morelenses whose group focal length is long but with individual elements ofstrong and opposite power. Thus, the astigmatism lens group may be madeof the same optical material without introducing significant lateralcolor. Since the beam size associated with any object point in space tobe imaged is quite small compared to the field of the beam, the strongelements tend to introduce deleterious amounts of spherical aberrationor coma into the final image.

The Objective Lens

The next component is a standard camera objective lens 230. The standardcamera objective lens 230 forms an image using theastigmatism-corrected, reflected light from the convex mirror 210. Inthe present embodiment, a standard off-the-shelf camera lens is usedthat is optimized for cost and performance in the conventionalphotography market. The current embodiment relies upon a pre-definedfocal length.

The focal length of the standard objective lens is selected based on twofactors. The first factor is the maximum angular field of view presentby the convex mirror and astigmatism correction lens group. This factoris determined by the largest angle away from the horizon of an object tobe captured. The second factor is the maximum diameter of the circularimage to be recorded. In an embodiment that uses 35 mm film, this valuewould not exceed 24 mm. In an embodiment that uses Charged CoupledDevice arrays, the objective lens must keep the circular image withinthe bounds of the CCD array.

For one preferred embodiment, the appropriate focal length is 22 mm.Since there are many objective lenses available with focal lengths inthe 18 mm to 24 mm range, this focal length provides many off-the-shelflens choices.

To allow a standard off-the-shelf camera lens to be used, the presentinvention “false focuses” the image beyond the normal focal plane. Thisallows the next optical element (field flattening lens) to fit betweenthe objective lens 230 and the image plane 250.

The Field Flattening Lens

Another optical problem created by the parabolic mirror is a curvedimage field that is created by the curve of the parabolic mirror. Thecurved image field problem is solved by adding yet another lens 240.This final lens is a “field flattening” lens, that flattens the field ofoptimal focus to a flat two dimensional image plane. The fieldflattening lens 240 must be kept as close to the image plane aspractical to eliminate the need for a focal plane shutter.

In one embodiment, the material SFL6 is used to create the fieldflattening lens 240. Due to its high index of refraction, SFL6 allowsthe field flattening lens 240 to be approximately 2 millimeters thick.If the field flattening lens 240 was created using more traditionalmaterials, the field flattening lens 240 would be approximately 4.5millimeters thick.

The Image Capture System

The final major component of the panoramic camera design is the imagecapture mechanism 250. The image capture mechanism 250 is placed at theimage plane just beneath the field flattening lens 240. This mechanismcaptures the optimized two dimensional annular image of the surroundingpanorama. An example of a captured panorama stored as a two dimensionalannular representation is shown in FIG. 3a.

In one embodiment of the present invention, the image capture mechanismcan be a frame of photographic film as illustrated in FIG. 4a. Usingconventional photography techniques, several successive frames can beused to record series of images. The series of images may be severaldistinct still images taken from different locations. Alternatively, theseries of images may be a set of successive images recorded used tocreate a panoramic motion picture. The image that is recorded ontophotographic film is then later converted into a digital image fordigital image processing as will be described in later sections of thisdocument.

In the preferred embodiment of the present invention, a high resolutiondigital image capture system is used to capture the annular imagecreated by the optical elements. In one embodiment of the presentinvention, a Charged Coupled Device (CCD) array 450 is placed in theimage plane to capture the image as illustrated in FIG. 4b. Controlcircuitry 460 coupled to the CCD array 450 captures the image directlyinto a digital format. The use of a digital image capture system allowsan immediate read-out of the digitized raw annular image. The digitizedraw annular image can be stored into a storage device 470 such as flashmemory or a hard disk drive. The CCD array 450 may be color (RGB) orblack & white, depending on the intended application.

To generate an annular image of sufficient quality to be used in theApple QuickTime® VR market, it has been determined that the image planemust be sampled with an array having at least 2K by 2K elements. To meetthis requirement, one embodiment of the present invention uses a CCDarray produced by Loral-Fairchild, Inc. However, the high resolution CCDarray sold by Loral-Fairchild, Inc., adds a significant cost to thepanoramic camera of the present invention. Furthermore, large CCD arrayssuch as the Loral-Fairchild array have difficulty handling the extremedifferences in light intensity that are produced by the optical systemof the present invention. Specifically, one area of the image may havedirect sunlight and other areas may receive comparatively little light.

To reduce the production cost of the panoramic camera, alternateembodiments of the present invention use a set of lower resolution CCDarrays. Specifically, consumer grade CCD devices that are targeted atthe consumer electronics market are used. Consumer electronics grade CCDarrays have the distinct advantages of lower cost, morehighly-integrated support circuitry availability, high speed read-out,robustness to extreme lighting and other environmental conditions.

No individual consumer grade CCD array meets the high resolutionrequirements needed by the present invention (at least 2K by 2Kelements). Therefore, a method of obtaining a greater image resolutionis required if consumer grade CCDs are used.

One method of creating an acceptable image capture mechanism usingconsumer grade CCD arrays is to use multiple low resolution CCD chips tocover the image plane using a mosaic pattern. Referring to FIG. 5, thebasic panoramic camera configuration described in the previous sectionsis illustrated except the last stage has an added beam-splitter 545 thatdirects a second image to a second field flattening lens 541 and asecond image plane 551. The beam-splitter 545 may comprise ahalf-silvered mirror or a prism arrangement as is known in the art. Thetwo image planes (image plane 551 and image plane 553) each capture aportion of the whole annular image. To construct a complete image, thecamera optically composites the images from the two different imageplanes into a single image.

FIGS. 6a and 6 b each illustrate one possible embodiment of the dualimage plane image capture system. FIG. 6a illustrates a mosaic patterncreated with four consumer grade CCD devices. As illustrated in FIG. 6a,the two image planes capture the whole annular image while each imageplane leaves room for the chip lead frames and support circuitry. FIG.6b illustrates an alternate embodiment that six consumer grade CCDdevices. An additional advantage of this scheme is that the CCD arraychips can potentially share some supporting circuitry since the signalseach independent chip requires are often identical.

A disadvantage of the mosaic technique is the image capture variationthat will exist between the different CCD chips. The image variation canbe compensated for by having overlapping CCD array coverage. Theoverlapping area is used to cross calibrate the image variation betweenadjacent CCD arrays.

Folded Optics Configuration

FIG. 7 illustrates an alternative embodiment of the panoramic camerasystem. In the embodiment of FIG. 7, the camera subassembly is housedwithin the parabolic mirror. Referring to FIG. 7, the convex mirror 710is inverted and a hole is cut into the tip. A second mirror 715 isplaced above the convex mirror 710, directing the light from thesurrounding panorama into the hole in the top of the convex mirror 710.The remainder of the optical path, including the astigmatism correctionlens 720, the objective lens 730, the field flattening lens 740, and theimage capture mechanism 750, are all housed inside the inverted convexmirror 710. It is apparent from the diagram of FIG. 7 that the “foldedoptics” configuration protects the optical path and mechanical parts ofthe panoramic camera system.

Transparent Block Configuration

Another alternative embodiment is shown in FIG. 8a. In this alternative,the convex mirror is formed as the internal space of a curved block oftransparent material such as glass or plastic. The mirror surface 810 isformed by the inner surface of a hole that is milled or cast in the topof the transparent material 805. The shape of the outer surfaceapproximates a sphere centered on the virtual focal point of the convexmirror 810. The outer surface of the transparent material is a polishedsurface that forms the outside skin of the camera. The bottom tip of thetransparent block is optically mated to the other optical parts of thecamera system. The bottom tip may be polished flat or molded into ashape that contributes to the astigmatism lens group.

The solid transparent block approach has a number of significantadvantages. First, the mirrored inner surface of the transparent blockmaterial can be well protected. This technique overcomes thedisadvantages of front surface mirrors. Specifically, when front surfacemirrors are exposed to the outside world they are susceptible to damageand degradation. In the above described embodiment, the mirrored surfaceis fully protected since it is encased between a protective backingmaterial and the transparent block material. Another advantage of thesolid block approach is that the skin of the camera is incorporated intothe optical system. Thus only one surface would need to be multicoatedto prevent internal reflections.

The transparent block technique can also be implemented using the foldedoptics scheme described in the previous section. Specifically, FIG. 8billustrates an inverted solid transparent block used to implement apanoramic camera system. In this case, the camera components arecontained within the mirror cavity. Note that the outside surface at thetop of the block is no longer an exit path but is instead a mirroredsurface that directs the image light down into an optical path insidethe parabolic block.

Different methods can be used to construct a transparent block panoramiccamera system. One method would be to create the transparent block, thenpolish the transparent block, and finally add a mirrored surface whereappropriate. An alternate method of constructing a transparent blockpanoramic camera system would start with the convex mirror. Then, theconvex mirror would be encapsulated within the transparent block. Thismethod would be simpler to construct since a concave surface would notrequire polishing. Furthermore, the convex mirror would be protected bythe transparent block.

Center Support Configuration

Another alternative embodiment addresses the problem of how to align andsupport the optical elements of the panoramic camera illustrated in FIG.2a. It is possible to use the protective, transparent block technique asdescribed in the previous section to provide structure, stability andalignment. However, the transparent block technique requiresmulticoating of the surfaces or else undesired internal reflections willbe visible. FIG. 9a discloses an alternate embodiment wherein a centralpost 903 is used to support the parabolic mirror 910. The remainder ofthe optical system is below the parabolic mirror 910 and the centralpost 903. The center support scheme takes advantage of the fact that thecenter of the annular image is discarded since it contains only an imageof the camera itself. Therefore, the center portion of the annular imagecan be used for support of the parabolic mirror 910.

External Support Configuration

Another scheme for supporting the parabolic mirror above the opticalelements below is to use several side supports. This can be accomplishedby splitting the parabolic mirror into “pie-pieces” by cutting theparabolic mirror after fabrication. For example, the parabolic mirrorcan be quartered as illustrated in FIG. 9b. The four sections ofparabolic mirror 1021, 1022, 1023, and 1024 can be spread apartslightly, allowing for the introduction of supporting elements 1031,1032, 1033, and 1034 that will not obstruct the fields of view.

If the parabolic mirror is split into four sections, then the annularimage will appear as four quadrants at the image plane. To correct forthis, the gaps can be removed during the polar-to-rectangular coordinateconversion, thereby restoring the continuity of the panoramic image. Thegaps between the mirror sections should be kept as small as possible,however, since the optical system is degraded by the loss of rotationalsymmetry.

Panoramic Image Presentation

As illustrated in FIG. 3a, the panoramic camera system of the presentinvention records a two dimensional annular representation of thesurrounding panorama. However, the annular representation is not of muchinterest to most viewers. Therefore, to display the panoramic imagescaptured by the panoramic camera of the present invention, severaldifferent display systems are disclosed.

Still image presentation as a rectangular panoramic image

The most common method of displaying a panoramic image is to display theimage as a rectangle where the horizontal direction represents the viewangle. An example of this type of panoramic image presentation isillustrated in FIG. 3b. Such rectangular panoramic images are commonlydisplayed in nature magazines. As stated in the background, the priorart method of creating such rectangular panoramic images was to takeseveral conventional photographs at different angles and then stitchthose photographs together somehow.

With the panoramic camera system of the present invention, suchrectangular panoramic images can easily be created. First, the panoramiccamera system of the present invention is used to capture an annularimage of the surrounding panorama. Then the annular image is digitizedand loaded into a computer system. (The image will already be in digitalform if a CCD version of the panoramic camera system was used to capturethe image.)

A custom conversion program is then executed on the computer system. Thecustom conversion program scans around the annular image starting at anarbitrarily chosen sampling line 310. Points along the sampling line 310are sampled and then their position changed using polar coordinate torectangular coordinate conversion. FIG. 10 illustrates how two differentpoints on the annular image are sampled and then placed into rectangularcoordinates. As illustrated in FIG. 10, the orientation of the samplingpattern changes as the coordinate transform program rotates around theannular image. The resulting rectangular image is illustrated in FIG.3b.

While sampling the annular image, it is important to sample the imagedifferently depending on where the annular image is being sampled. Thefollowing three rules must be observed:

1. The sampling shape is dynamically changing depending on the viewingangle (both in the horizontal and vertical).

2. The sampling shape size is proportional to the radius (verticalviewing angle); and

3. The sampling shape orientation is different depending on thehorizontal viewing angle.

Since there is a greater resolution around the outer perimeter of theannular image, the corresponding rectangular image portion will havebetter image clarity. The outer perimeter of the annular image may bethe top or the bottom of the rectangular image depending on the opticalpath. (Compare FIG. 2a with FIG. 7). In FIG. 10, the lower portion ofthe rectangular image will have a better image clarity since it is fromthe outer perimeter of the annular image. One embodiment of the presentinvention takes advantage of this fact by using the outer perimeter ofthe annular image for the ground since the ground in a panoramic sceneis generally more detailed than the sky.

Once the panoramic image has been converted from an annular image to arectangular image on a computer system, then the rectangular image canbe presented to viewers in a number of different formats. For example,the rectangular image may be distributed electronically as a JPEG imageand viewed with JPEG image viewers. Alternatively, the rectangular imagecan be printed out with a color printer. It should be noted that sincethe rectangular image is in digital form, it can quickly be added to apublication being created with a Desktop Publishing Layout Program suchQuarkXpress or Adobe's PageMaker.

Image presentation as a Virtual Reality image

Apple Computer introduced a standard known as QuickTime® VR for storingand displaying virtual reality images. Apple Computer's QuickTime® VRstandard governs the data storage format and the playback softwareneeded to view the QuickTime® VR datasets. The camera system of thepresent invention can be used to quickly create QuickTime® VR datasets.

The QuickTime® VR format stores the image as cylindrical image asillustrated in FIG. 11a. Specifically, the viewpoint is at the center ofthe cylinder and the inner surface of the cylinder represents the storedQuickTime® VR image. Note that trapezoid shaped patches must be sampledto generate an image if the user is looking up or down as illustrated inFIG. 11b.

FIG. 11c illustrates a flow diagram that lists the steps required toproduce a QuickTime® VR dataset using the panoramic camera system of thepresent invention. First, at step 1110, a panoramic image is recordedwith the panoramic camera. Then, at step 1120, the recorded image isdigitized and loaded into a computer system. If the panoramic camerarecorded the image on a piece of film, then a print of the film can bescanned into the computer system using a flatbed scanner. Alternatively,a film image can be commercially transformed into the well knownPhotoCD® format produced by Kodak®. If the panoramic camera recorded theimage with a CCD array and stored the image digitally, then the digitalimage is just copied from the camera's storage system into the computersystem's storage system.

After the digital version of the annular image is available on thecomputer system, a transformation program is then executed on thecomputer system at step 1130 in order to transform the digitized annularimage into a QuickTime® VR dataset. The annular image produced by thecamera system of the present invention stores the panoramic imageinformation in a polar coordinate system. Conversely, Apple®'sQuickTime® VR uses a cylindrical coordinate system as illustrated inFIG. 11a. Thus, the transformation program converts the annular imagefrom its polar coordinate system into the QuickTime® VR cylindricalcoordinate system. After transforming the image into the QuickTime® VRcylindrical coordinate system, then a file is created using theQuickTime® VR file format at step 1135.

Once the coordinate transform is complete, the transformed image can beviewed using Apple's QTVR player program as stated in step 1140.

Still image presentation on a computer network

Since the present invention can store the annular image in digital form,a very useful method of distributing panoramic images is through acomputer network. In particular, the hypertext transport protocol (http)of the World Wide Web (WWW) on the Internet can be used to distributestill annular images. The still annular images would be stored on aWorld Wide Web server. To access the still annular images, any usercoupled to the Internet would use a World Wide Web browser program.

One method of transporting the images would be to define a new panoramicimage annular data format. The images could then be downloaded as storedin the panoramic image annular data format. A helper application wouldthen display the images once downloaded.

A better method of displaying images using the hypertext transportprotocol (http) of the World Wide Web (WWW) would be to implement a“plug-in” application that would work with the browser program. FIG. 12aillustrates a graphical user interface for one possible client panoramicimage presentations system. On the right side of FIG. 12a, a set ofdifferent panoramic images to display is available. To display one ofthose images, the user selects the image with a cursor control device.On the upper left of the graphical user interface of FIG. 12a isviewport for displaying a portion of a panoramic image. Pan arrows oneither side of the viewport allow the user to pan left and right.

Image Presentation As Video

One of the most interesting presentation systems for the presentinvention is a video presentation system. FIGS. 13a and 13 b illustrateone possible video presentation system.

Referring to FIG. 13a, a CCD version of the panoramic camera system 1205of the present invention is illustrated coupled to a computer system1200. The CCD version of the panoramic camera system 1205 is coupledthrough a panoramic camera interface 1210. The panoramic camerainterface 1210 receives a digital stream of annular images. To interfacewith computer systems, one embodiment of the panoramic camera interface1210 is the FireWire system that is described in the IEEE 1394 standard.

After being received through the panoramic camera interface 1210, thedigitized annular images are stored in an Annular “Video” Storage system1230. The Annular “Video” comprises a series of a consecutive annularimages taken with a CCD version of the panoramic camera system 1205.

To display the Annular Video as normal video, the annular frames must beconverted from the annular image format into normal video images. In oneembodiment of the present invention, only a portion of the annular imageis converted into normal video. One reason for this is that the aspectratio of video does not allow for good viewing of wide but shortrectangular panoramic images. Furthermore, by only transforming aportion of the annular image into normal video, the transformation canbe done in real-time without requiring exceedingly fast computerequipment. The transformation of annular video to normal video is doneby annular to video conversion units 1240 and 1243.

To display the normal video, existing video streaming software 1260 and1263 can be used. For example, using a standard transmission protocollike MPEG or proprietary protocols such as StreamWorks produced by XingTechnology Corporation of Arroyo Grande, Calif., or VDOLive produced byVDOnet Corporation of Santa Clara, Calif., the video can be provided tocomputer users coupled to a network.

FIG. 13b illustrates one possible embodiment of a graphical userinterface (GUI) for accessing the annular video. In the GUI of FIG. 13b,a video viewport 1340 is used to display the video. A smaller stillpanoramic image 1310 is used to illustrate a static version of the fullpanoramic video. A locator window 1315 is used to identify the viewangle that the video window 1340 is displaying within the full panoramicview that is available.

To change the view angle, the user can select a pan right arrow 1347 ora pan left arrow 1343 with a cursor 1320. Alternatively, the user cansimply move the position of the locator window 1315 within the stillpanoramic image 1310. In the embodiment of FIG. 13b, the entire verticalimage aspect of the image is compressed into the video viewport 1340.

Referring back to FIG. 13a, user input processing routines 1250 and 1253processing the user's commands. When the user requests a viewpointchange, the new viewpoint is communicated to the respective annular tovideo conversion units 1240 or 1243 such that it will begin convertingimages from the new user viewpoint. In an alternate embodiment, the userinput processing routines are placed within the client program on theclient computer system. For example, in an embodiment of a WWW browserprogram, a plug-in program can process the user commands and simply passthe location of the video viewport to the server.

Referring back to FIG. 13b, a parameter window 1350 is also available tothe viewer. The parameter window 1350 allows the user to adjust some ofthe viewing parameters such as Image Brightness 1352, Image Tint 1353and Image Contrast 1355. When a user adjusts these parameters, thechanges will be processed by the user input processing routines 1250 and1253 provided to the annular to video conversion units 1240 or 1243 orthe video streaming software 1260 or 1263 such that video quality ischanged.

Telepresence: Video and Audio

To more completely convey the experience of being at a differentlocation, the present invention can be combined with a three dimensionalsound system. Referring to FIGS. 14a and 14 b, an embodiment of thecamera system is illustrated with four directional microphones 1441,1442, 1443, and 1444. The four directional microphones 1441, 1442, 1443,and 1444 capture sound emanating from four cardinal directions.

To add three dimensional sound, the sound from the various directionalmicrophones is mixed depending on the viewing angle that a user hasselected. For example, if a viewer that is seeing a real-time image fromcamera 1400 of FIG. 14a is viewing straight out of the page, then theleft speaker will receive information from microphone 1443 and the rightspeaker will receive information from microphone 1441. The sound can beprovided to users on a computer network using audio streaming softwaresuch as RealAudio by Progressive Networks, Inc. As the viewer adjuststhe viewing angle, the sound from the directional microphones will beadjusted accordingly.

By adding sound to the system, the user is provided with cues as towhich direction they should be viewing. For example, if the user hears asound from “behind”, then the user can change the view angle to lookbackward.

The foregoing has described a camera device that captures 360 degreepanoramic images and presentation systems for displaying such images. Itis contemplated that changes and modifications may be made by one ofordinary skill in the art, to the materials and arrangements of elementsof the present invention without departing from the scope of theinvention.

We claim:
 1. An apparatus for capturing panoramic images, said apparatuscomprising: a convex mirror, said convex mirror reflecting light fromsubstantially 360 degrees around said convex mirror and from afield-of-view extending through a horizon line, said convex mirrorhaving a substantially parabolic shape; an astigmatism correction lens,said astigmatism correction lens correcting astigmatism in said lightreflected from said convex mirror, said astigmatism resulting from theshape of said convex mirror; an objective lens for focusing said lightreflected from said convex mirror as corrected by the astigmatismcorrection lens; and an image capture mechanism, said image capturemechanism capturing said light reflected from said convex mirror,whereby said light is captured linearly proportional to the incidentangle of said light on said mirror.
 2. An apparatus for capturingpanoramic images, said apparatus comprising: a convex mirror, saidconvex mirror reflecting light from substantially 360 degrees around theconvex mirror; at least one correction lens, the at least one correctionlens correcting at least one optical aberration in said light reflectedfrom the convex mirror, said at least one optical aberration a result ofthe convex mirror; and at least one image capture mechanism configuredto capture a portion of said light as corrected by the at least onecorrection lens.
 3. The apparatus of claim 2 wherein said at least oneoptical aberration is astigmatism and the at least one correction lensis an astigmatism correction lens configured to correct saidastigmatism.
 4. The apparatus of claim 2 wherein said at least oneoptical aberration is field curvature and the at least one correctionlens is a field flattening lens configured to flatten said fieldcurvature to a field of optimal focus.
 5. The apparatus of claim 3wherein said at least one optical aberration includes field curvatureand another one of the at least one correction lens is a fieldflattening lens configured to flatten said field curvature to a field ofoptimal focus.
 6. The apparatus of claim 2 wherein said light is from aband of light around the convex mirror.
 7. The apparatus of claim 6wherein said band of light around the convex mirror extends through ahorizon line.
 8. The apparatus of claim 6 wherein: said light reflectedfrom the convex mirror forms an annular image; and the convex mirror isshaped such that an angle of elevation from said horizon line islinearly proportional to a radial distance of the center of said annularimage.
 9. The apparatus of claim 2 wherein the at least one imagecapture mechanism comprises photographic film.
 10. The apparatus ofclaim 2 wherein the at least one image capture mechanism comprises anelectronic image capture device.
 11. The apparatus of claim 10 whereinsaid electronic image capture device comprises at least one chargedcoupled device.
 12. The apparatus of claim 10 further comprising: a beamsplitter, the beam splitter dividing said light into a first image and asecond image; wherein the at least one image capture mechanism comprisesa first electronic image capture mechanism and a second electronic imagecapture mechanism; and wherein said first electronic image capturemechanism captures at least half of said first image and said secondelectronic image capture mechanism captures at least half of said secondimage.
 13. The apparatus of claim 2 further comprising: a protectivetransparent block, the protective transparent block covering the convexmirror.
 14. The apparatus of claim 2 further comprising: a secondmirror, the second mirror reflecting said light reflected from theconvex mirror through an aperture in the convex mirror; and wherein theat least one image capture mechanism captures said light reflected fromthe second mirror.
 15. The apparatus of claim 14 wherein the at leastone image capture mechanism is housed within the convex mirror.
 16. Theapparatus of claim 14 further comprising: a protective transparentblock, the protective transparent block covering the convex mirror. 17.The apparatus of claim 16 wherein the protective transparent block endsat said aperture in the convex mirror.
 18. The apparatus of claim 16wherein a lens system is housed within an indentation of the protectivetransparent block.
 19. The apparatus of claim 16 wherein the protectivetransparent block has an external surface and said external surface isshaped such that light rays that will be reflected onto the at least oneimage capture mechanism enter the protective transparent block at anormal to said external surface.
 20. The apparatus of claim 2 whereinsaid light is captured as an annular image, and said apparatus furthercomprises a transformation mechanism configured to transform a portionof said annular image.
 21. The apparatus of claim 20 wherein thetransformation mechanism comprises a computer configured to perform saidtransformation.
 22. The apparatus of claim 20 wherein the transformationmechanism comprises circuitry configured to perform said transformation.23. The apparatus of claims 2, 3, 4 or 5 further comprising an objectivelens configured to focus said portion of said light onto the at leastone image capture mechanism.
 24. The apparatus of claims 2, 3, 4 or 5wherein said portion of said light reflected from the convex mirrorforms an annular image.
 25. The apparatus of claims 2, 3, 4, 5, 12, 13,14 or 19 wherein said portion of said light captured by the at least oneimage capture mechanism represents a capture angle of at least 45degrees of said substantially 360 degrees of said light reflected by theconvex mirror.
 26. The apparatus of claim 25 wherein said capture angleis substantially 180 degrees.
 27. An apparatus for capturing panoramicimages, said apparatus comprising: a convex mirror, said convex mirrorreflecting light from substantially 360 degrees around the convexmirror; an astigmatism correction lens, the astigmatism correction lenscorrecting astigmatism in said light reflected from the convex mirror,said astigmatism a result of the convex mirror; an objective lens, theobjective lens focusing said light as corrected by the astigmatismcorrection lens onto at least one image capture mechanism; and a fieldflattening lens, the field flattening lens correcting field curvature insaid light from the objective lens to flatten said field curvature to afield of optimal focus at said at least one image capture mechanism,said field curvature a result of the convex mirror.
 28. A method forrecording a panorama comprising steps of: reflecting light from asubstantially 360 degree band around a convex mirror to an image plane;correcting at least one optical aberration in said light reflected fromsaid convex mirror by at least one correction lens, said at least oneoptical aberration a result of said convex mirror; and capturing aportion of said light as corrected by said at least one correction lensby at least one image capture mechanism, wherein said at least oneoptical aberration includes astigmatism and said at least one correctionlens is an astigmatism correction lens configured to correct saidastigmatism.
 29. The method of claim 28 wherein said at least oneoptical aberration includes field curvature and another one of said atleast one correction lens is a field flattening lens configured toflatten said field curvature to a field of optimal focus.
 30. The methodof claim 28 further comprising focusing said portion of said light,corrected by said at least one correction lens, onto said image planewith an objective lens.
 31. A method for recording a panorama comprisingsteps of: reflecting light from a substantially 360 degree band around aconvex mirror to an image plane; correcting at least one opticalaberration in said light reflected from said convex mirror by at leastone correction lens, said at least one optical aberration a result ofsaid convex mirror; and capturing a portion of said light as correctedby said at least one correction lens by at least one image capturemechanism, wherein said at least one optical aberration is fieldcurvature and said at least one correction lens is a field flatteninglens configured to flatten said field curvature to a field of optimalfocus.
 32. The method of claims 28, 29 or 31 wherein said portion ofsaid light as corrected by said at least one correction lens representsa capture angle of at least 45 degrees of said substantially 360 degreesof said light reflected by the convex mirror.
 33. The method of claim 32wherein said capture angle is substantially 180 degrees.
 34. The methodof claim 32 wherein said portion of said light as corrected by said atleast one correction lens forms an annular image.
 35. The method asclaimed in claim 34 further comprising transforming a portion of saidannular image.
 36. The method of claim 32 wherein said portion of saidlight is captured on a photographic film.
 37. The method of claim 32wherein said portion of said light is captured by at least one chargedcoupled device.
 38. A method for capturing panoramic images comprisingsteps of: reflecting light from a substantially 360 degree band around aconvex mirror to an image plane; correcting astigmatism in said lightreflected from said convex mirror, said astigmatism a result of saidconvex mirror; focusing said astigmatism corrected light onto at leastone image capture mechanism; and correcting field curvature in saidlight from the objective lens to flatten said field curvature to a fieldof optimal focus at said at least one image capture mechanism, saidfield curvature a result of the convex mirror.